fix(claude-channel): benign dedup for shared-agent reply race#1309
fix(claude-channel): benign dedup for shared-agent reply race#1309sh1nj1 wants to merge 6 commits into
Conversation
Two Claude Code channel sessions in the same working directory share one Collavre agent (default AGENT_NAME). Both stream from agent:user:<id>, so a single work-topic dispatch fans out to both and each session's turn calls /reply with the same task_id. The server is designed to dedup these via the atomic task claim (claim_delegated_task), returning 409 to the loser. But resolve_reply_agent scoped its task lookup to status: "delegated". Once the winning session flipped the task to "done", the loser's lookup returned nil and reply rendered a misleading 403 "Not authorized" — which the MCP plugin threw as a hard error and surfaced to the user. Server: resolve_reply_agent no longer gates on status. It resolves the agent for any task on the topic owned by the caller's Claude Channel agent and lets claim_delegated_task decide actionability — so the loser reaches the intended 409. Nonexistent/foreign task_ids still 403 (ownership + topic scope unchanged). Plugin: reply() treats 409 as a benign dedup (handled:false) instead of throwing; the reply tool reports "already answered by another session" rather than a failed reply.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 90efe44df7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (res.status === 409) { | ||
| const respBody = await res.text(); | ||
| return { handled: false, reason: respBody }; |
There was a problem hiding this comment.
Do not swallow every reply conflict as dedup
When the original dispatch is cancelled or failed instead of answered (for example, offline-session cancellation or stuck-task recovery flips a delegated task while Claude is still composing), /reply now also produces a 409 because the server resolves non-delegated tasks and the atomic claim fails. This branch treats every 409 as a benign sibling-session dedup, and index.ts then reports “Already answered” without posting any comment, so a valid response can be silently dropped. Please distinguish the completed/dedup case from cancelled/failed/not-delegated conflicts before suppressing the error.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 96bb8ac. The claim fails identically in both cases, which is exactly why the client could not tell them apart: claim_delegated_task returns nil for any task that is not delegated, so a dispatch cancelled by an offline session, failed, or flipped by stuck-task recovery while Claude was composing produced the same bare 409 as the sibling-session dedup. The dedup case has an answer already posted; the others have none, and the text in hand is the only copy of it — so index.ts reporting "Already answered" and posting nothing lost a real reply, silently.
The server now names the cause rather than leaving the client to guess:
TaskClaimService#conflict_reason→already_completedwhen the task isdone(a sibling won the claim, or the agent's own job completed it — either way the dispatch was answered and this reply is a duplicate),not_delegatedfor every other status.replyrenders it beside the existing message:{ error: ..., reason: ... }, still 409.
The client suppresses on the marker only. isBenignReplyDedup parses the body and requires reason === "already_completed"; anything else — not_delegated, an unparseable body, or a response with no reason at all — is thrown as Reply failed (409). That last one is the back-compat call: an older server sending the unmarked body gets the pre-dedup behaviour (a visible error) rather than the silent drop. Surfacing a conflict the model can act on is recoverable; dropping the user's answer is not.
CONFLICT_UNKNOWN_TASK is deliberately not a third reason. resolve_reply_agent resolves the agent from the task and 403s if it is not this caller's on this topic, so by the time the conflict branch runs the task always exists — a find_by miss in conflict_reason would be unreachable, and it folds into not_delegated (surface it) as the safe default.
Tests pin both directions. Ruby: an answered dispatch replied to twice returns already_completed; cancelled, failed and pending tasks each return not_delegated with an explicit "suppressing this 409 loses the reply" assertion. Making conflict_reason unconditionally return already_completed fails that second test. TypeScript: isBenignReplyDedup over the dedup body, the cancelled body, a legacy no-reason body, and non-JSON.
agents_controller_test 74 runs / 340 assertions / 0 failures; plugin suite 32/32; tsc --noEmit and rubocop clean.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
A 409 from /agent/reply has two causes. The dedup one — a sibling session sharing this agent won the atomic claim and posted — is benign. The other is a task that left `delegated` WITHOUT being answered (offline-session cancel, failure, stuck-task recovery mid-turn); the claim fails identically, but nothing was posted, so swallowing it drops the only copy of the reply. The server now names the cause: TaskClaimService#conflict_reason returns already_completed for a done task and not_delegated for everything else, and reply renders it alongside the error. The client suppresses only on the explicit already_completed marker — an unmarked body, including one from an older server, is raised as before dedup existed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 96bb8ac0eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| def conflict_reason(agent:, topic:, requested_task_id:) | ||
| task = Task.find_by(id: requested_task_id, agent_id: agent.id, topic_id: topic.id) | ||
| task&.status == "done" ? CONFLICT_ALREADY_COMPLETED : CONFLICT_NOT_DELEGATED |
There was a problem hiding this comment.
Only mark conflicts benign after a reply exists
When the first sibling has claimed a delegated task, claim sets the task to done before comment.save/finalize links the reply; if that worker crashes or the save later fails (for example blank/invalid text, after which the controller restores the task to delegated), a concurrent valid reply can hit this method during the done-without-reply window and receive reason: "already_completed". The plugin now suppresses exactly that reason, so the valid answer is dropped even though no comment was ever created; base the benign reason on an actual linked reply (or otherwise wait for finalization), not status alone. Fresh evidence here is the new status-only already_completed marker.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 03f5a1d. status == "done" was standing in for "the dispatch was answered", and it is not the same claim: #claim flips the task to done, and the controller saves the comment after that returns. Inside that window the row reads completed with nothing posted behind it — and the window can end in a rollback (comment.save false → the controller restores the task to delegated) or never end at all if the worker dies. A concurrent /reply arriving there was told already_completed, so the client suppressed it and dropped the only copy of an answer nobody else had written.
The linked reply comment is the proof, since #finalize is what creates the link (comment.update_column(:task_id, task.id) → Task#reply_comment):
return CONFLICT_NOT_DELEGATED unless task&.status == "done"
task.reply_comment.present? ? CONFLICT_ALREADY_COMPLETED : CONFLICT_CLAIMED_WITHOUT_REPLYA third reason rather than folding it into not_delegated: a claimed-but-unanswered task really is done and really did leave delegated, so reporting not_delegated would put a false statement in the operator-facing half of the payload for the sake of the machine-readable half. The client keys on the one explicit already_completed marker, so a new value is non-benign by construction and needs no client change to be safe — but the test pins it anyway, because "unknown reasons happen to fall through to safe" is not something the next edit should be free to break.
One deliberate consequence: two replies that genuinely race — the second arriving in the microseconds between the winner's claim and its save — now surface a conflict instead of staying quiet, and the caller sees an error for a duplicate. That is the trade this direction has to make. Nothing at 409 time can distinguish "the winner is mid-save" from "the winner died mid-save"; only one of those two mistakes loses a message, and a retry of the spurious one converges (by then the reply is linked, and the second attempt is suppressed as dedup).
Tests both ways: a task set to done with no linked comment — exactly what #claim leaves behind — now answers claimed_without_reply, and reverting the predicate to the status check fails it with already_completed. The plugin side pins that the new reason is not benign.
agents_controller_test 75 runs / 344 assertions / 0 failures; plugin node --test 33/33; rubocop clean on both changed files.
conflict_reason read `status == "done"` as "the dispatch was answered", but #claim flips the task to done BEFORE the controller saves the comment. Inside that window the row says completed with nothing posted behind it, and the window can end in a rollback (invalid text restores the task to `delegated`) or never end at all (the worker dies). A concurrent /reply arriving there was told already_completed and the client dropped it — the one copy of an answer nobody else had written. The linked reply comment is what #finalize creates, so its presence is the only proof an answer exists. already_completed now requires it; a claimed but unanswered task reports claimed_without_reply, which the client does not suppress. Reasons stay diagnostic — the client suppresses on the one explicit marker, so the new value needs no client change to be safe.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 03f5a1da02
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # `reply_comment` is what #finalize links (comment.task_id = task.id), so | ||
| # its presence is the only proof the dispatch was actually answered. | ||
| task.reply_comment.present? ? CONFLICT_ALREADY_COMPLETED : CONFLICT_CLAIMED_WITHOUT_REPLY |
There was a problem hiding this comment.
Wait for finalization before hard-failing duplicate replies
When two same-directory Claude sessions reply at nearly the same time, the loser can reach this check after the winner's claim has committed status = done but before finalize has linked the reply comment. In that normal sibling-race window reply_comment.present? is still false, so the server returns claimed_without_reply; the client then throws instead of treating the race as benign, leaving the original 409-as-user-visible-error behavior timing-dependent even though the winning reply is about to land. Consider waiting/reloading for finalization (or otherwise making claim+link atomic) before deciding this is a hard conflict.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 97e3bda. You are right that the previous round made the loser's outcome timing-dependent, and the direction of that dependence is the bad one: the sibling race is the common case, and it lands in the done-without-reply window every single time. #claim commits done, then the controller runs comment.save — validations, callbacks, broadcasts — and only then does #finalize reach comment.update_column(:task_id, ...). Every winning claim passes through claimed-without-reply on its way to answered, so deciding on the first read was reporting "your answer may be the only copy" for what is almost always a duplicate that is milliseconds from landing.
The predicate is unchanged — only an actually linked comment is ever called benign, and nothing is inferred from status or elapsed time. What changed is that finalization gets a bounded chance to produce the proof before the answer is fixed:
FINALIZE_GRACE_SECONDS = 0.5
FINALIZE_GRACE_INTERVAL = 0.05
deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + FINALIZE_GRACE_SECONDS
loop do
return true if reply_linked?(task)
return false if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline
sleep FINALIZE_GRACE_INTERVAL
endMonotonic clock so a wall-clock adjustment cannot stretch or collapse the deadline mid-wait. The reads are uncached — that one is load-bearing rather than defensive: ActiveRecord::QueryCache wraps the request, so the identical SELECT 1 FROM comments WHERE task_id = ? would be served from the first (empty) result for the entire wait and the loop could never observe the winner's commit.
Only the window pays for it. A non-done task returns not_delegated before the wait is reached, and a reply that was already linked when the request arrived answers on the first read — both pinned by tests that assert the elapsed time stays under the grace period, so a later edit cannot quietly put the common paths behind a sleep.
I did not take the make-claim-and-link-atomic option, though it is the tighter one on paper. Doing it means holding the SELECT FOR UPDATE across comment.save, which is exactly what the current shape avoids: the long comment above #claim exists because that save fires after_update_commit work — TriggerLoopCheckJob, the stop-button broadcast — and running it inside the claim transaction puts those side effects before the commit they depend on. update_all is there specifically to keep them out. Moving the claim after the save instead reopens the duplicate-comment race the claim was written to close. Waiting buys the same observable behaviour for the loser (it blocks either way, on a lock or on a poll) without moving side effects inside a transaction.
What survives past the deadline is the case you and I both want surfaced: a claim whose worker died mid-save. It reports claimed_without_reply 500ms later than before, and nothing suppresses it.
Four tests in a new task_claim_service_test.rb, and reverting the wait to a single read fails two of them: a link that lands between the first and second read now reads as already_completed (with an assertion that it re-read at all, so a first-read decision fails even if it guesses right), and the never-finalized claim asserts both that the window was actually given and that it is bounded.
task_claim_service_test 4 runs / 22 assertions / 0 failures; agents_controller_test 75 runs / 344 assertions / 0 failures; full pre-push suite 2946 runs / 9267 assertions / 0 failures; rubocop clean on both files.
The winning claim commits `done` a whole comment save before #finalize links the reply, so every ordinary sibling race passes through claimed-without-reply on its way to answered. Deciding on the first read made the loser's outcome a function of how fast the winner's INSERT was: a hard 409 the model has to recover from, or silence, depending on timing. conflict_reason now polls for the link for up to 500ms before answering. The proof is unchanged — only an actually linked comment is ever called benign — the wait just gives finalization the moment it needs to produce one. A claim that never finalizes (worker died mid-save) still surfaces, one deadline later. Reads are uncached: Rails' per-request QueryCache would otherwise serve the first empty SELECT for the whole wait. Paths with nothing to wait for skip it entirely: a non-done task answers not_delegated immediately, and an already-linked reply answers dedup on the first read.
Problem
Running two
claude --channels plugin:collavre@collavresessions in the same working directory and sending a message in a Collavre topic: both sessions receive the dispatch, the first replies fine, the second fails with:Root cause
Same-directory sessions share one Collavre agent (the default
AGENT_NAMEcase). Both subscribe toagent:user:<id>, so a single work-topic dispatch fans out to both and each session's turn calls/replywith the sametask_id. This is by design —ClaudeChannelAdapterdocuments that a work topic "may be handled by any session (the server's atomic task claim dedups concurrent handlers)", with the loser getting a clean 409.But
resolve_reply_agentscoped its task lookup tostatus: "delegated". Once the winning session'sclaim_delegated_taskflipped the task todone, the loser's lookup returnednilandreplyrendered a misleading 403 at the auth layer — one step before the intended 409 dedup. The MCP plugin throws on any non-2xx, so the 403 surfaced to the user as a hard error.The existing test
concurrent replies for same task_id only one wins; loser gets 409even admitted this in its assertion (weakened torefute 2xxwith a comment noting the real result was 403).Fix
Server (
resolve_reply_agent): drop thestatus: "delegated"filter. Resolve the agent for any task on the topic owned by the caller's Claude Channel agent, and letclaim_delegated_task(which keeps the atomicWHERE status='delegated') decide actionability. The loser now reaches the designed 409. Nonexistent / foreign-agent / wrong-topictask_ids still 403 (ownership + topic scoping unchanged).Plugin (
collavre-client.ts+index.ts):reply()treats 409 as a benign dedup (handled: false) instead of throwing; the reply tool reports "already answered by another session" rather than a failed reply.Verification
engines/collavre/test/controllers/api/v1/agents_controller_test.rb— 69 runs, 0 failures (updated the concurrent-reply test to assert:conflict).tscclean build +npm test(28 pass).🤖 Generated with Claude Code